A scalable MPI implementation of an ad-hoc parallel I/O system
نویسندگان
چکیده
This paper presents the novel design, implementation and evaluation of an ad-hoc parallel I/O system (AHPIOS). AHPIOS is the first scalable parallel I/O system completely implemented in Message Passing Interface (MPI). The MPI implementation brings the advantages of portability, scalability and high-performance. AHPIOS allows MPI applications to dynamically manage and scale distributed partitions in a convenient way. The configuration of both MPI-IO and storage management system is unified and allows for a tight integration of the optimizations of these layers. AHPIOS partitions are elastic: they conveniently scale up and down with the number of resources. We develop two collective I/O strategies, which leverage a two-tiered cooperative cache in order to exploit the spatial locality of data-intensive parallel applications. The file access latency is hidden from the applications through an asynchronous data staging strategy. The two-tiered cooperative cache scales with both number of processors and storage resources. Our experimental section demonstrates that with various optimizations integrated AHPIOS offers a substantial performance benefit over the traditional MPI-IO solutions on both PVFS or Lustre parallel file systems. Index Terms Parallel I/O, Parallel Systems, Distributed File Systems, Parallelism and Concurrency.
منابع مشابه
A Scalable Message Passing Interface Implementation of an Ad-Hoc Parallel I/o system
In this paper we present the novel design, implementation, and evaluation of an ad-hoc parallel I/O system (AHPIOS). AHPIOS is the first scalable parallel I/O system completely implemented in the Message Passing Interface (MPI). The MPI implementation brings the advantages of portability, scalability and high performance. AHPIOS allows MPI applications to dynamically manage and scale distribute...
متن کاملThe Impact of File Systems on MPI-IO Scalability
As the number of nodes in cluster systems continues to grow, leveraging scalable algorithms in all aspects of such systems becomes key to maintaining performance. While scalable algorithms have been applied successfully in some areas of parallel I/O, many operations are still performed in an uncoordinated manner. In this work we consider, in three file system scenarios, the possibilities for ap...
متن کاملUsing IKAROS to provide Scalable I/O bandwidth
We present IKAROS as a utility that permit us to form scalable storage platforms. IKAROS enable us to create ad-hoc nearby storage formations and use a huge number of I/O nodes in order to increase the available bandwidth. We measure the performance and scalability of IKAROS versus the IBM’s General Parallel File System (GPFS) under a variety of conditions. The measurements are based on benchma...
متن کاملDesign and evaluation of two scalable protocols for location management of mobile nodes in location based routing protocols in mobile Ad Hoc Networks
Heretofore several position-based routing protocols have been developed for mobile ad hoc networks. Many of these protocols assume that a location service is available which provides location information on the nodes in the network.Our solutions decrease location update without loss of query success rate or throughput and even increase those.Simulation results show that our methods are effectiv...
متن کاملDesign and evaluation of two scalable protocols for location management of mobile nodes in location based routing protocols in mobile Ad Hoc Networks
Heretofore several position-based routing protocols have been developed for mobile ad hoc networks. Many of these protocols assume that a location service is available which provides location information on the nodes in the network.Our solutions decrease location update without loss of query success rate or throughput and even increase those.Simulation results show that our methods are effectiv...
متن کامل